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Environment 
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Abstract 

In this exploratory experimental study, 135 pre-service teachers developed an instructional plan for a case study within the 
MIMIC (Multiple Intelligent Mentors Instructing Collaboratively) computer-based environment. Three-dimensional, animated 
pedagogical computer agents, representing constructivist and instructivist approaches to instructional planning, served as 
instructional mentors within the environment and were available to provide suggestions. The research design was comprised of 
two two-factor MANOVAs with the instructivist agent (present, absent) and constructivist agent (present, absent) serving as the 
two factors, with two groups of dependent measures —awareness and attitude. Additionally, the value of the agents and overall 
differences between high and low performers were investigated. I^garding awareness, main effects for the presence of the 
constructivist agent indicated that when the constructivist agent was present participants tended to report a change in their 
perspective of instructional planning, reflected less on their thinking, and developed instructional plans rated as more constructivist 
in underlying pedagogy. Regarding attitude, a main effect for the presence of the instructivist agent indicated that when the 
instructivist agent was present, participants reported a more negative disposition regarding instructional planning. Results are 
discussed in terms of the impact on teaching instructional planning to pre-service teachers. 

Introduction 

In the field of instructional design, there are diverse theories and approaches to instruction (e.g., (Driscoll, 2000)). For pre- 
service teachers, the importance of seeing how these theories relate to real instructional problems is critical. Two prominent yet 
differing approaches to instructional planning are systematic instructional planning (referred to here as instructivism), based on 
an objectivist epistemology, and constructivism, based on an interpretivist epistemology (Jonassen, 1991). These two 
philosophical approaches lead to different understandings of human cognition and affect both the instruction that is developed 
and what evaluations are feasible and appropriate (Roblyer, 1996; Yarusso, 1992). 

With its objectivist epistemic roots regarding knowledge, the underlying assumption of traditional instructional planning is 
that knowledge can and should be transmitted from teacher to student. An instructivist approach to instructional planning 
emphasizes knowledge transfer and teacher- centered learning environments, where skills are taught sequentially, incorporating 
individualized work with traditional assessment methods (Roblyer, Edwards, & Havriluk, 1997). This type of systematic 
approach to instruction has been shown to be effective due to its focus on clearly identifying goals and systematically developing 
instructional activities and assessment that lead to the attainment of the goals (Reiser & Dick, 1996). 

In contrast, constructivism has its epistemic roots in interpretivism, which maintains that knowledge is personally constructed 
within individuals and does not exist external to the individual. The constructivist approach tends to focus on more student - 
centered environments, to provide activities that facilitate knowledge construction and generative learning (e.g., Wittrock, 1990). 
Driscoll (2000) describes five attributes of constructivist instruction: 1) embedding learning in complex and realistic 
environments; 2) providing for social negotiation; 3) supporting multiple perspectives and use of multiple modes of 
representation; 4) encouraging ownership in learning; and, 5) nurturing self-awareness of the knowledge construction process 
(pp. 382-383). To implement these features as part of the constructivist planning process, pre-service teachers must learn to 
emphasize the process of learning more than the end product. Constructivist approaches have been found to be particularly 
beneficial for developing meaningful learning activities and engaging students in hi^er order thinking (Jonassen, Peck, & 
Wilson, 1999) 

One way to authentically demonstrate these two distinct approaches to pre-service teachers would be through seasoned 
professionals modeling the approaches in the context of a real instructional situation. Exposure and interaction with several 
experts describing instructional content matter from different points of view can be very rewarding for the learner (Laurel, Oren, 
& Don, 1990) and can help the learner to establish the best personalized approach to understanding the content. Further, such 
exposure to multiple pedagogical perspectives could enhance pre-service teachers’ cognitive flexibility by requiring them to 
independently consider alternative points-of-view. Viewing an instructional problem from multiple perspectives is also desirable 
for promoting reflective thinking and problem solving, qualities important for pre-service teachers who are learning to be 
teaching professionals. Further, as Jonassen (1997) describes, “instructional planning is an archetypal ill-structured problem 
because “the designer is constrained by circumstances, though in most design problems, there are a variety of solutions, each one 
of which may work as well as any other (p. 69).” Given that more than one problem-solving path is possible to reach a solution, 
the ability for a pre-service teacher to take multiple perspectives on instructional planning is appropriate and necessary. Overall 
while it may be beneficial for our pre-service teachers to see their role in the classroom from multiple pedagogical perspectives 
(Bennett & Spalding, 1992), devising this sort of experiential exposure is difficult to implement with human instructors. 
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A promising possibility for demonstrating and experiencing different instructional approaches is through computer-based 
agents serving as a pedagogical mentors (Baylor, in press). A software agent is an independent computer program operating 
within software environments such as operating systems, databases, or computer networks (Roesler and Hawkins, 1994). Agents 
appear to have the characteristics of an animate being, and simulate a human relationship by doing something that another person 
could otherwise do for you (Seiker, 1994). Animated pedagogical agents have lifelike qualities, and can employ verbal 
instructional explanations together with nonverbal forms of communication (e.g., gaze, gesture, conveying emotion) in 
interacting with the learner. Along this line, Lester and colleagues (Johnson, Rickel, & Lester, 2000; Lester, Stone, & Stelling, 
1999) have suggested that life-like agent characters are ideal to serve as tutors, coaches or guides in knowledge-based learning 
environments. 

Further, learners treat computer-based agents as human, even when the computer interface is not explicitly anthropomorphic 
(Reeves & Nass, 1996). 

Building upon Laurel’s (1990; 1997) suggestion for agents to represent different “roles” such as characters in a play, the next 
question to consider is whether agents could represent different instructional roles as pedagogical mentors. While the idea of 
representing multiple instructional roles through computer-based media has been implemented in other research, there have been 
limited controlled studies. The ETOILE system for teaching educational psychology principles (Dillenbourg, Mendelsohn, & 
Schneider, 1994) incorporated five agents, each labeled after the teaching styles they implement: Skinner, Bloom, Vygotsky, 
Piaget, and Papert. Each agent was implemented as an independent rule base that was separated out from the content rather than 
being domain-specific. The five teaching agents implement decreasing level of directiveness: Skinner works step by stop, Bloom 
makes larger steps but with close control of mastery, Vygotsky is based on participation, Piaget intervenes only to point out 
problems and Papert does not interrupt the learner. The ETOILE system also includes a “coach” agent that is in charge of which 
tutor is used although the learner may also select or remove a tutor. The ETOILE system was not designed for the purpose of 
instructional research, but rather to conceptualize the underlying engineering principles for the multiple agents; consequently, 
there is no empirical evidence regarding its instructional impact. 

MIMIC (Multiple Intelligent Mentors Instructing Collaboratively) is an agent-based learning environment developed for the 
purpose of instructional research (Baylor, 1999, 2000b). MIMIC situates instructional planning within a specific context: a case 
study of a thirteen-year old girl struggling with the economics concepts of supply and demand. In MIMIC, agents explicitly 
represent different perspectives of instructional planning (objectivism and constructivism) and facilitate pre-service teachers’ 
internalization of these approaches. The animated three-dimensional pedagogical agents serve as scaffolds, providing cognitive 
support to pre-service teachers while they write an instructional plan. The learner has control over the amount of assistance and 
when it will be provided by the agent(s). In related research, participants working with both the instructivist and constructivist 
pedagogical agents could differentiate between them and could explicate the two theories that they represented (Baylor, in press). 
The agents were also reported to be believable and useful. (Baylor, in press). 

Hietala & Niemirepo (1998) suggest that the same social factors that occur in learning communities with human beings are 
also influential in a learning community consisting of multiple artificial teaching and learning agents. They refer to this aspect as 
the need for pedagogical multiplicity of teachers, suggesting that the many levels and complexities of the learning process might 
be alleviated by providing more alternatives to the learner via an "extended family of intelligent agents." Essentially, an agent- 
based learning environment such as MIMIC allows pre-service teachers to figuratively “put on different hats” and facilitates them 
in switching roles when needed to solve an instructional problem. Through experiential interaction with the agents, the pre- 
service teacher is facilitated in a deep approach to the task, focusing on the meaning of the instructional planning process itself, 
rather than a surface approach which would involve simply writing an instructional plan following a “recipe.” In this way, the 
pre-service teachers' experience of instructional planning and the specific meaning it has for them could be considered as the 
most fundamental aspect of learning (Marton & Booth, 1997). 

In this exploratory experimental study, it is hypothesized that the presence of the instructivist and constructivist agent(s) 
(especially both simultaneously) will impact learners’ awareness of instructional planning by increasing their reflection during 
the instructional planning process and changing their perspective regarding instructional planning. It is not hypothesized that the 
presence of agents will affect performance given that it is generally found that the presence of agents does not significantly 
improve performance (Dehn & van Mulken, 2000); however, it is predicted that pre-service teachers’ transformation of 
awareness will be reflected by the instructivist or constructivist “flavor” of their instructional plans, depending on which agent(s) 
are present. In terms of attitudes, it is predicted that the presence of the agents as scaffolds would positively influence 
participants* dispositions, self-efficacy and perceived instrumentality regarding instructional planning. 

Methods 

Sample 

The sample consisted of 135 pre-service teachers, in eight sections of an “Introduction to Educational Technology” course. 
As part of this required course, the participants had already been taught an instructivist model of instructional planning (Reiser & 
Dick, 1996) and a constructivist approach to instructional planning (Grabe & Grabe, 2001) with identical course material (e.g., 
lectures, Powerpoint slides, assignments, exams) across the eight sections. Participation in this study was a required activity for 
class participants, and they received course credit for participating. The mean age of the sample was 19.76 years (5D=2.13). Of 
those reporting ethnicity, 84% were Caucasian, 4% were Hispanic, 10% were African American, and 2% were of other groups. 
Of those reporting gender, 21.5% of the sample were male and 78.5% were female. Sixty percent (the majority) of the 
participants were sophomores, 27% were juniors with 7% freshman and 6% seniors. In terms of prior experience with 
instructional planning, participants’ mean score was 2.23, p/>=.97), where l=no experience and 5=very much experience. 
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indicating that overall they had little prior experience. There were no significant differences in age and GPA among the 
participants in all conditions. In terms of ethnicity, gender, and year in school, chi-square analyses revealed no significant 
differences between the groups. 

Multiple Intelligent Mentors Instructing Collaboratively (MIMIC) Environment 

From the learner’s perspective, MIMIC consists of an introduction, a case study, blueprints stage, plan stage, and assessment 
stage. The introduction begins with the statement that “We are pleased that you have decided to join our educational consulting 
firm, ‘Instruction Inc.’ Given your new skills in instructional planning, we have a project for which we really need your help,” 
and briefly describes the case study situation with thirteen-year-old Anna and her teacher Mr. Lange. Following this, the 
participant is instructed how to move throughout the environment: 

Through our computer-based system, you will be able to follow 3 steps in devising your instructional solution. The three 
components of the project include the following: 

1. STEP 1: Blueprints for Instruction: This is where you will first determine the goals of the instruction; in other words 
WHAT you want the learner to be able to do at the end of the process. 

2. STEP 2: Developing the Instructional Plan: Here you will get into the "heart" of the planning process, and develop a 
detailed plan that a teacher can use to carry out the instruction as you specify. 

3. STEP 3: Assessment: At this stage you will determine how to measure whether the learner actually learned what you 
intended. 

At any time you may refer back to the details of the case study to refresh your memory of the situation. 

You may go back and forth as much as you would like among the 3 components. And when you are completely finished with 
your plan, press the Finished button you'll find in the assessment stage. 

Next, if participant is in a 1- or 2-agent condition, the personal Advisor(s) (see "Pedagogical Agents" section) would 
introduce themselves and their role. 

The environment organized the participant’s instructional planning processes into four main phases which will be described 
below, each indicated through large icon-buttons: case study, blueprints, planning, and assessment. At any time it is possible for 
the participant to move from one phase to the other although it is not possible for the participant to return to the introductory 
screens. Once the participant enters the assessment stage, an additional button labeled “Finished” is provided. After selecting 
“Finished” the participant is asked “Are you ready to exit the application and go to the exit survey?” Upon selecting “OK” the 
participant answers post-questions. 

Case study. The case study was developed for MIMIC given that it is difficult to find existing case studies that are 
appropriate (Ertmer & Russell, 1995). It consisted of a description of Anna and her problems learning supply and demand, her 
teacher Mr. Lange, and her school in Texas. The concept of supply and demand was chosen as it is relatively domain- 
independent of specialty areas for instruction and requires less specific prior knowledge. Links were provided so that the 
participant could access Anna's homework that contained comments from Mr. Lange, and his personal planning notes which 
included text and graphics. In this way, participants could review the necessary content for themselves as well as evaluate Anna’s 
situation. 

Blueprints. The purpose of this phase was listed on-screen as follows: “The purpose of this step is for you to decide what you 
want Anna to learn. What have you determined to be the learning goals? List them clearly in the workspace below. For reference 
you may want to see the stated Texas standards and benchmarks regarding supply/demand for eighth graders, with links below.” 
A text -box field was provided within which the participant could list the instructional goals or objectives. Two links provided 
additional information regarding Texas standards and benchmarks for supply/demand. 

Planning. The purpose of this phase was listed on-screen as “To develop a detailed instructional plan for Anna. “ A text -box 
field was provided within which the participant could enter the instructional plan. 

Assessment. The purpose of this phase was listed on-screen as follows: “The purpose of this phase is to develop ways to 
determine if Anna learned what you initially defined in the blueprints phase. Please describe this assessment in detail in the space 
below. “ A text -box field was provided within which the participant could list the assessment. 

Pedagogical Agents 

Depending on the experimental condition, one or two Microsoft Agent characters (Peedy the Parrot and Merlin the Wizard) 
were implemented as “Advisors” to the participants. Characters were randomly assigned to represent the instructivist and 
constructivist agents and to control for possible differences. The Advisors were referred to by gender-neutral names - Jan and 
Chris. Jan was always the instructivist advisor, representing traditional systematic instructional planning including the problem- 
solving aspects of Instructional Systems Design (ISD) as characterized by Dick & Carey (1996) and Reiser & Dick (1996). Chris 
was always the constructivist advisor, representing a learner-centered approach, focusing on the importance of the context of 
learning, stressing that learning involves active interaction, aid emphasizing the process rather than the product of learning 
(Driscoll, 2000). 

The purpose of the agents was to serve as mentors (Baylor, 2000a) and to operationalize the instructivist and constructivist 
approaches to instructional planning. When one or two agents were present the following events resulted: 1) the agent(s) 
provide(s) an initial observation upon entering each of the four MIMIC planning stages; 2) the agent(s) provide(s) reflection 
questions to encourage self-evaluation consisting of statements “Make sure you are not Just talking about how you would do it; 
actually create the instruction for Mr. Lange (Anna’s teacher).”, “Actually develop the content-related activities”, or “ Apply the 
plan specifically to the topic of supply and demand” every five minutes upon entering a stage; 3) the agent(s) would provide an 
example of their instructional plans following the participant’s development of an instructional plan; and, 4) the agent(s) would 
provide additional suggestions when selected by the participant Agent suggestions were specific to the case study and were 
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developed and validated by experts in instructional planning together with the consultation of an economics professor. The 
available suggestions (specific to each planning phase) would appear in a pop-up box for the participant to select. For example, in 
the plan phase, one available suggestion is “What is my role in the learning process?” If this suggestion were selected from the 
instructivist agent, the agent would reply “You need to be in charge of the learning process for Anna. You need to organize the 
materials for Anna, to create an optimal learning environment.” The same suggestion if requested from the constructivist agent 
would be “Anna should be at the center of the learning process. This will encourage Anna’s initiative, get Anna to think and to 
reflect, and make the information real for Anna.” The blueprints phase had two suggestions, the plan phase had five suggestions, 
and the assessment phase had one suggestion. See a related study (Baylor, in press) for a complete listing and description of all 
agent suggestions. 

Measures 

Awareness. Awareness was assessed through three dependent measures: whether the participant changed perspective in 
instructional design, amount of participant’s reflective thinking, and the underlying pedagogy of the participant -designed 
instructional plan. To assess whether use of the system changed participants' perspective of instructional design, they were first 
asked “Did using this program change your perspective of instructional design? (yes/no)" which was coded as a 1 (yes) or 0 (no). 
To assess participants’ self reported reflections, they were asked, “How often did you reflect on your thinking during the 
process?” on a Likert scale of 1-3 with “Not at all,” “Several times,” and “Frequently” as the three levels. To assess the 
underlying pedagogy of the instructional plans, the instructional plans were scored according to their overall pedagogical 
“flavor,” on a scale from 1 to 10. Given that certain instructional plan features are representative of both instructivist and 
constructivist pedagogies (e.g., the importance of considering prior knowledge), the focus was on assessing the presence (or 
absence) of constructivist characteristics, as they were more salient and differentiable. A high score in this measure indicates that 
there were more constructivist aspects to the plan such as a student -centered approach, students’ involvement with constructing 
knowledge, a focus on students’ reasoning/critical thinking, and/or situated learning. A low score in this measure indicates that 
there are less constructivist and more characteristically instructivist elements within the plan. Two of the researchers met and 
together discussed what characterized a score of 1-10 for the presence of underlying pedagogy (where l=not at all constructivist 
and I0=highly constructivist) for five sample instructional plans. Following that, each researcher independently scored 15 
instructional plans. Inter-rater reliability between the two researchers was determined to be greater than .9 for the fifteen 
instructional plans. One of the researchers then scored the remainder of the instructional plans using the same rubric. Both 
researchers were blind as to the conditions of the participants throughout the rating process. 

Attitudes. Attitudes were assessed by three dependent measures, each as repeated-measures: self-efficacy, disposition, and 
perceived instrumentality. To assess participants’ self efficacy, one item to measure the students' self-efficacy beliefs about 
instructional planning was administered before entering and after exiting the MIMIC environment. It was developed based on 
Bandura and Schunk's (1981) guidelines. All participants were asked "How sure are you that you can write a lesson plan?" on a 
scale from 1 being not-sure to 9 being very sure. The test-retest reliability was r=.62 (/?<.001). To assess participants’ disposition 
toward instructional planning, each participant was asked to write two adjectives to "Describe what you think about instructional 
planning." This method was employed to obtain the participants' personal affect regarding instructional planning as opposed to 
the response set that could bias them to choose more favorable adjectives if adjectives were presented in a list. The adjectives 
were coded according to three levels: as -1 if both were negative, as 0 if one was negative and the other positive, and as +l if both 
were positive. The items were coded by two raters independently. Interrater reliability was established at .95. There were only 
two disagreements about two sets of adjectives which were resolved through discussion. Two adjective pairs were discarded 
because they could not be classified. The validity of this measure was established in (Kitsantas & Baylor, 2001) through 
concurrent validity of initial disposition with initial self-efficacy scores, given that research has shown that self-efficacious 
students generally have positive affect (Bandura, 1986). The test-retest reliability was r=.55, /?<.001. To assess the participants' 
perceived instrumentality, or perceived importance of instructional planning, the participants were asked to rate "How important 
is writing a lesson plan to you as a future professional?" on a scale of 1 to 5 where 1= not important, 2=fairly important, 
3=important, 4=very important, and 5=extremely important. Test-retest reliability was r=.83,/?<.001. 

Performance. Within MIMIC, all participants developed an instructional plan to teach the concepts of supply and demand to 
Anna. Each instructional plan was scored according to a rubric that consisted of four sub-areas. The four sub-areas of the rubric 
were goals/objectives, procedure, assessment, and holistic, the first three being aligned with the major components of 
instructional planning (goals/objectives, procedure, and assessment). For the goals/objectives sub-score, the plans were rated 
according to how clearly the goals/objectives were stated and how specifically the purpose of instruction was described. For the 
procedure sub-score, the plans were rated according to the meaningfulness and effectiveness of the instructional activities, 
whether they were in a logical sequence, and whether they addressed the goals stated in the blueprints phase. For the assessment 
sub-score, the plans were rated according to whether the assessment matched the goals/objectives, and whether it was logical. For 
the holistic sub-score, the plans were rated according to whether the plan was overall reasonable and effective. The overall 
performance score was the compilation of these four sub-scores (each rated from 1 to 5), with a potential range of 4-20. Two of 
the researchers met and together discussed what characterized a score of 1 through 5 (where l=poor and 5=excellent) for each of 
the four sub-areas for five sample instructional plans. Following that, each researcher independently scored 15 instructional 
plans. Inter-rater reliability between the two researchers was determined to be greater than .9 for the fifteen instructional plans. 
One of the researchers then scored the remainder of the instructional plans using the same rubric. Both researchers were blind as 
to the conditions of the participants throughout the rating process. 
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A 2 ent value. Participants in conditions where agents were present were asked to rate the value of the agents in several areas. 
Specifically, they were asked via Likert-scale formatted questions, “Did you enjoy working with <agent>?” (Not at all / A little / 
Very much/ Extremely); “Did you pay attention when <agent> made suggestions?” (Not at all / Not usually / Usually / Always); 
“Overall, was <agent> annoying or useful?” (Extremely annoying / annoying / useful / very useful); and, “Did you generally 
agree with <agent>’s suggestions? (yes/no)” 

Procedure 

All participants logged in to the MIMIC computer environment and answered computer-based questions regarding gender, 
age, and class section number. Next, the participants' perceived instrumentality, disposition regarding instructional planning, 
prior experience with instructional planning, and self-efficacy beliefs toward instructional planning were assessed. Following 
these initial measures, the participant entered the introduction to the MIMIC environment (see the MIMIC section). Following 
this introduction, and immediately before entering the environment, participants’ self-efficacy regarding the project was 
ascertained. Next, the participants worked through the case study, blueprints stage, planning stage, and assessment stage, 
developing an instructional plan. Depending on the condition (see “Pedagogical Agents” section), 0-2 agents were present within 
the environment, serving to represent instructional planning approaches (objectivism and/or constructivism). All participants 
worked independently within the environment at their own pace. Following completion of the instructional plan within the 
environment, all participants answered computer-based questions regarding amount of self-reflection, value of agent(s), 
perspective of instructional planning, perceived instrumentality, disposition, and self-efficacy. The entire procedure took 
approximately 90 minutes. 

Design and Data Analysis 

A three-factor MANOVA (instructivist agent : present, absent; constructivist agent : present, absent; agent character : Peedy, 
Merlin) was the initial method used for data analysis where agent character (Peedy the Parrot or Merlin the Wizard) was assigned 
as a within-subjects factor to test for possible differences in agent character. After it was determined that agent character did not 
play a factor, that factor was removed from further analysis leaving a two-factor MANOVA as the main method for data analysis. 
The data was analyzed according to two groups of dependent measures: awareness (comprised of change in perspective, 
reflection, and underlying pedagogy of instructional plan), and attitude (comprised of self-efficacy, disposition, and perceived 
instrumentality), each of which was assessed via a two-factor MANOVA. The analysis of attitude (comprised of self-efficacy, 
disposition, and perceived instrumentality) was treated as a repeated measures MANOVA. For some analyses that focused on 
agent combinations, a one-way MANOVA/ANOVA was also performed with condition (no agents, instructivist only, 
constructivist only, both agents) as the factor. To analyze participants’ value of the agents, independent -group t-tests were used 
for the questions comparing one agent on a particular attribute, and paired-group t-tests or chi-square analysis were used for the 
questions regarding participants who received both agents. 

Results 

General 

There were no statistically significant differences among agent conditions regarding the average number of suggestions 
requested from the agents. The average number of suggestions selected was M=10.34. 

Awareness 

The three dependent measures for awareness include whether the participant changed in perspective regarding instructional 
planning, amount of self-reflection, and underlying pedagogy of participant-designed instructional plan and were assessed 
through a two-way MANOVA with instructivist (present, absent) and constructivist (present, absent) as the be tween-subject 
factors. 

Perspective of instructional planning. Results from the two-way MANOVA revealed a main effect for the constructivist 
agent on change in perspective in instructional planning, F(l, 131) = 9.82,/?<.01 , where M=J\ (present) versus M=AS (absent), 
indicating that when the constructivist agent was present, participants were more likely to report that MIMIC changed their 
perspective of instructional planning. There were no other significant main effects or interactions. To determine the relative 
differences of change in perspective among the four agent conditions, a one-way ANOVA with condition (no agents,- instructivist 
only, constructivist only, both agents) as the factor was conducted and revealed a significant main effect for agent condition, F(3, 
131)=3.74, p=.01. Post-hoc Tukey’s tests indicated that the constructivist-only condition elicited the most change in perspective 
(A/=.80) and was significantly greater than both the instructivist-only condition (M=.47) and the no-agent conditions (M=.44) , 
but was not significantly greater than the both-agents condition, which ranked second in overall change in perspective (Af=.63). 

Self-reported reflection. The two-factor MANOVA revealed a main effect for the constructivist agent, indicating that when 
the constructivist agent was present, participants reported reflecting less 0/=2.24), than when it was absent 0/=2.43), F(l, 
131)=4.73,p<.05. There were no significant differences between high and low performers on self-reported reflection. 

Underlying pedagogy of instructional plan. The two-factor MANOVA indicated a main effect for the constructivist agent, 
F(l,131) =1 1.28, /?=.001, where the presence of the constructivist agent was related to participants developing more 
constructivist-oriented instructional plans (V^=6.12) than when it was absent (V^=4.47). While there was not a statistically 
significant main effect for the instructivist agent, there were numerical differences showing that its presence was associated with 
lower scores ^=4.92) than its absence 0/=5.72), indicating its positive relation to an instructivist underlying pedagogy. An 
independent t-test also showed that high achievers developed plans that were significantly more constructivist 0/=6.O8) in 
approach than low achievers (A/=4.29). The overall mean for the underlying level of pedagogy for all participants was A#=5.35, 
5D=3.01. 
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Attitude Attitude was analyzed through a repeated-measures two-factor MANOVA, with disposition, self-efficacy, and 
perceived instrumentality as the dependent measures, each assessed both before and after working in MIMIC, and with the 
instructivist agent and the constructivist agent as the two between-subject factors. 

Disposition regarding instructional planning. The two- factor repeated-measures MANOVA indicated a main effect for the 
presence of instructivist agent F(l, 128)=3.46, p<.05, revealing that when the instructivist agent was present, participants had 
significantly lower dispositions regarding instructional planning (M=.54) than when the instructivist agent was absent (A/=.75). 
Overall, participants’ disposition toward instructional planning was generally positive (where -1 represents negative, 0 represents 
neutral, and 1 represents positive), both before (M=.63) and after (M=.58) working within MIMIC. 

Selfiefficacy. The two-factor repeated-measures MANOVA revealed a main effect for time of assessment, showing that 
participants overall increased in self-efficacy (A/=5.46 vs. Af=6.01) as a result of MIMIC, F( 1,1 28)= 12. 54, p=.001. There were 
no significant main effects or interactions for the presence or absence of the constructivist and instructivist agents. As would be 
expected, self-efficacy was highly related to participants' prior experience with instructional planning. Those participants with 
high experience writing lesson plans (3, 4, 5) as compared to those with low experience (1,2; where l=no experience and 5=very 
much experience), had significantly higher self-efficacy both before 0f=6.4O vs. Af=4.91), /(1 33 ) = 4.06, /K.OOl, and after 
(M=6.62 vs M=5.63), <133) =3.11, p=.00 1 , using MMIC. 

Perceived instrumentality. The two-factor repeated-measures MANOVA did not show any significant main effects or 
interactions. Participants’ perceived instrumentality was M=4.11 for the pre-test, and A^4.18 for the post-test, indicating that 
they believed instructional planning to be slightly more important than “very important” both before and after the use of MIMIC. 

Performance Performance was analyzed through a two- factor MANOVA, with each of the four sub-scores and the total 
score as the dependent measures, with instructivist (present, absent) and constructivist (present, absent) as the two between- 
subject factors. Results revealed no main effects or significant interactions for the total performance score or each of the four sub- 
scores. The total performance score ranged from 4-20, with the overall average of Af=13.71, 5Z)=4.10. Descriptive statistics for 
the sub-scores were as follows: goals/objectives — Af=3.38, SD=\. 32\ procedure— Af= 3.30, SD 1.10; assessment — A#=3.26, 
5D=1.29; and, holistic - A/=3.23, SZ>=1.14. Based on the total performance score, participants were categorized as high 
performers if they scored in the top quartile (total score of M=\6 and above), and low performers if they scored in the bottom 
quartile (total score of A/=10 and below). 

Agent value Agent value was assessed in several areas, as listed below. 

Enjoyment in working with agent. Participants tended to enjoy working with both agents (Af= 2.63 for instructivist agent and 
A/=2.64 for constructivist agent), where 2=”A little” and 3=”Very much.” A non-significant t-test of the two groups (those with 
constructivist present versus those with instructivist present) revealed that participants reported finding both agents to be equally 
enjoyable. Attending to agent. Participants tended to pay attention to both agents {M= 3.05 for instructivist agent and Af=3.21 for 
constructivist agent) where 3= “usually” and 4=”always”. A non-significant t-test of the two groups revealed that participants 
reported paying relatively equal attention to both agents. Usefulness of agent. Participants tended to report that both agents were 
useful 0f=3.O2 for instructivist agent and Af=3.08 for constructivist agent) where 3=”useful” and 4=”very useful”. A non- 
significant t-test of the two groups revealed that participants reported finding both agents equally useful. Agreement with agent. 
In terms of whether participants agreed with the agents’ advisements, answers were coded as yes=l and no=0, and results show 
that participants tended to agree with both of the agents’ suggestions, M=.61 for instructivist and A/=.84 for constructivist. A non- 
significant t-test of the two groups revealed that participants reported not agreeing significantly more or less with either agent. 

Discussion 

Overall, the results indicate that the presence of the constructivist pedagogical agent affected pre-service teachers’ 
metacognitive awareness of instructional planning in multiple ways: through a change in perspective, less reported reflection, and 
through the underlying pedago^ of their instructional plans. 

It is speculated that increased awareness about instructional planning would lead to a richer and more comprehensive 
understanding of the planning process, leading pre-service teachers to develop an appreciation for the process. As stated by 
Marton and Booth (1997), “of prime interest is the variation in the ways in which people are capable of experiencing various 
situations or phenomena. If one becomes aware that something is in a certain way, they also become aware that it could be in 
some other way” (p.207). Eventually, it would be expected that this change in perspective and understanding of the depth and 
complexity of instructional planning could lead to better performance and/or increased intrinsic motivation related to the task. 

The presence of the constructivist agent tended to change participants’ perspective toward instructional planning. Although 
participants had been introduced to the constructivist approach as part of the course in which they were enrolled, it still may have 
been experienced as a novel and unique approach. Given that some pre-service teachers describe instructional planning 
negatively, using adjectives such as “boring,” or “tedious” (Kitsantas & Baylor, 2001), the constructivist approach may have 
been perceived as offering something new and providing more options for instructional planning. Further, the presence of the 
constructivist pedagogical agent could have been perceived to highlight more appealing elements of instructional planning (such 
as a student -centered focus or responsibility of the learner). 

While the two-agent condition (given that it had the constructivist agent present) was more transforming than the instructivist 
only or no agent conditions, it was not found to lead to a significantly greater change in perspective than the other three agent 
conditions, as was predicted. The fact that the presence of two agents simultaneously was not perceived as the most transforming 
in terms of a change in perspective could be an issue of cognitive load. As Sweller and colleagues suggest (Sweller, van 
Merrienboer, & Paas, 1998) “less is best” in learning situations, indicating that in this case the learners may be too focused during 
problem solving to process suggestions from multiple agents. 
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The finding that the presence of the constructivist agent led to reflection seems at first incompatible with the finding that 
the constructivist agent led to a greater change in perspective. However, when the constructivist agent was present, perhaps 
participants were focusing their attention on its ideas/suggestions rather than reflecting on their own cognitive processes. In other 
words, it seems viable that the presence of constructivist agent facilitated pre-service teachers to think more (i.e., change 
perspective), but not necessarily to reflect more. While there is strong evidence that reflection during instructional activities is 
important (Chi & VanLehn, 1991; VanLehn, Jones, & Chi, 1992), there is less information regarding the relative value of 
reflection as compared to awareness. Future research should include an open-ended follow-up question to determine what pre- 
service teachers actually meant bv reporting less reflection. 

The presence of the constructivist agent was also associated with participants’ developing more constructivist-oriented 
instructional plans, reflecting a “trickle down” effect of the agent's pedagogical beliefs to the participants. Although there was not 
a main effect where the presence of the instructivist agent was related to lower underlying pedagogy score (thus indicating a more 
instructivi St-oriented underlying pedagogy in the instructional plans), there were numerical differences showing that the presence 
of the instructivist was associated with lower scores in this area, indicating that in both cases (presence of instructivist and 
presence of constructivist) the pre-service teachers internalized the agent’s suggestions and translated them in their instmctional 
plans. 

Overall, the agents were perceived by the pre-service teachers to be valuable as mentors. Participants reported neither agent to 
be “better” or “worse” in any of the following aspects: enjoyment of working with agent; paying attention to agent; perceived 
usefulness of agent; and, credibility of agent. Further, the pre-service teachers who received both agents were equally split as to 
which agent made them think the most, which thought the most like them, and which of the two agents they would choose to 
assist them (if could choose only one). Agents were rated as equally useful, they also were paid equal attention, and were equally 
enjoyable with which to work. 

While there were no explicit differences between the agents in terms of value, there were differences in the effect of the agents on 
attitude. Contrary to what was hypothesized, it was found that the presence of the instructivist agent led participants to report 
significantly lower dispositions regarding instructional planning. Given that the instructivist agent represents a systematic 
approach, perhaps students felt it was too prescriptive, and made the instruction and/or planning process seem tedious or boring. 
While there were no main effects on self-efficacy for the presence of the agents, self-efficacy increased for participants using the 
MIMIC system, suggesting a practice effect. A reason that may explain why there were no effects of the agents on perceived 
importance of instructional planning could be that participants started with already -high ratings (rating it on average as slightly 
more important than “very important”). 

There were no main effects of the agents’ presence for the overall performance score which was expected given that related 
research has not provided evidence that animated agents improve learning (Dehn & van Mulken, 2000). Further, the MIMIC 
agents provided suggestions regarding the underlying pedagogic rationale for different aspects of the planning process, not 
solutions. While these advisements were content-specific, they did not specifically prescribe or show the students exact 
implementation. 

In terms of the overall implications, there is preliminary evidence to suggest that the exposure to constructivism as an 
instructional planning process adds richness, diversity, meaning and interest. While the instructivist approach adds substance and 
structure to the process, it may negatively affect disposition. If only one perspective could be provided to enrich their awareness 
of instructional planning, exposure to the constructivist approaches may be most beneficial, especially if the pre-service teachers 
already have a strong foundation in instructivist approaches to instructional planning. 

Future research could implement the study with more advanced pre-service teachers or instructional designers, to determine 
how the agents impact them in terms of awareness of and attitude towards instructional planning. Another way to explore a 
change in awareness could be through including epistemology profiles to determine if pre-service teachers epistemic beliefs 
change as a result of using the system. The role of reflection needs to be further investigated through more open-ended questions 
and to systematically evaluate the agents for their self-regulatory features to determine what promotes monitoring and evaluation 
and how they relate to what the participant terms “reflection.” Cognitive load as an explanation for the impact of two agents 
needs to be further examined with more advanced students, who may be able to better manage receiving advisements from 
multiple agents. 

Overall, this study validated the effectiveness of an agent -based approach as a research process to investigate teaching and 
learning by simulating human-like mentoring via pedagogical agents (Baylor, in press). The instructivist and constructivist 
agents within MIMIC provided an indirect and meaningful way to investigate students’ affect and beliefs toward instructional 
planning. 
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